Apparatus and method for predicting biometrics based on fundus image

ABSTRACT

Provided are apparatus and method for predicting biometrics using a fundus image. The method for predicting biometrics using a fundus image includes steps of preparation of a plurality of learning fundus images, generation of a learning model for predicting corresponding biometrics using the prepared data based on at least one characteristic of the fundus reflected in the prepared plurality of learning fundus images, reception of a prediction target of fundus image, and prediction of the biometrics of the subject of the prediction target of fundus image by using the generated learning model.

BACKGROUND Field

The present disclosure relates to an apparatus and a method forpredicting biometrics based on a fundus image. Particularly, the presentdisclosure relates to an apparatus and a method for predictingbiometrics such as the axial length, the fundus age, and the like, whichare important for early diagnosis of ophthalmic diseases and braindiseases, using a fundus image and a deep neural network-based learningmodel.

Description of the Related Art

In an examination process in an ophthalmology department, a fundus image(fundus photo) is highly useful in that the fundus image is non-invasiveand can be easily obtained with a basic examination. Particularly, thereare advantages in that the fundus image may be taken with little effortand time even for patients who are relatively uncooperative or patientswith decreased eyesight and may be easily photographed even in a statein which the pupil of a subject is not dilated.

However, in the related art, an expensive fundus imaging device wasrequired to acquire a fundus image, but recently, there has beenprovided a technique capable of easily acquiring a fundus image using alens mounted on a smartphone without an expensive device for fundusimaging.

In addition, the fundus image may be utilized to detect various retinaldiseases. For example, in the diagnosis of various ophthalmic diseasessuch as diabetic retinopathy, retinal vein occlusion, and maculardegeneration, the opinions of a specialist based on the fundus image arerequired.

In particular, the axial length related to the fundus condition of thepatient is an important factor in evaluating the patient's condition,such as determining myopia or retinal disease. Specifically, it has beenknown previously that blood vessels of the retina and the choroid, thecolor of the background fundus, and the like are related to the axiallength, but a technique capable of accurately performing numericalprediction for the axial length using this has not been known.

In addition, dedicated measuring equipment that has been developed tomeasure the axial length up to now corresponds to expensive and advancedequipment. In countries that cannot afford such measuring devices, or donot have developed medical infrastructure, although it is necessary tomanufacture an intraocular lens of an appropriate size for the treatmentof ocular diseases such as cataracts occurring in elderly patients,there is a limitation in that it is difficult to acquire detailedinformation on the axial length of a patient.

In addition, it is known that it is possible to determine a vascularcondition of the retina from the condition of the fundus and apossibility of developing diseases related to cardiovascular diseasesand brain diseases. When the fundus age that comprehensively reflectsthe fundus condition of the patient is calculated numerically, it isexpected that a systemic blood vessel condition, a brain aging risk, andthe like of the patient are inferred by comparing the actual age of thepatient with the calculated fundus age to be used as an indicator forthe patient's health condition.

The background art of the present disclosure is disclosed in KoreanPatent Publication No. 10-2019-0087272.

SUMMARY

An object to be achieved by the present disclosure is to provide anapparatus and a method for predicting any type of biometric of a subjectusing his or her fundus image that can be easily captured, in thepurpose of reducing any type of cost to measure the biometrics such aspreparing an expensive equipment or time and space required to operatethe measuring machine, and the biometrics include the axial length, thefundus age, and the like.

Objects of the present disclosure are not limited to the above-mentionedobjects, and other objects, which are not mentioned above, can beclearly understood by those skilled in the art from the followingdescriptions.

According to an aspect of the present disclosure, there is provided amethod for predicting biometrics using a fundus image including steps ofpreparing a plurality of learning fundus images, generating a learningmodel based on at least one characteristic of the fundus reflected inthe prepared learning fundus images, receiving a prediction target offundus image, and predicting the biometrics of the subject.

The biometrics may include an axial length.

At least one characteristic of the fundus reflected in the fundus imagesmay include the shape of the choroidal vessels and the optic disc, andany condition of the fundus shown in a fundus image.

The method for predicting the biometrics based on the fundus imageaccording to an exemplary embodiment of the present disclosure mayfurther include producing the evidence area in a prediction target offundus image, which indicates the contribution to the prediction of thebiometrics expressed as numerical values greater than or equal to apredetermined threshold.

The method for predicting the biometrics using the fundus imageaccording to an exemplary embodiment of the present disclosure mayfurther include outputting emphasized evidence area in a predictiontarget of fundus image.

The preparation of a plurality of learning fundus images may includeacquiring not only a plurality of learning fundus images as the input, acorresponding biometric data as the label to train a learning model, butalso the age data of the subjects of a plurality of learning fundusimages, as an additional information for the training.

The generation of the learning model may include generating a learningmodel for predicting the biometrics based on at least one characteristicof the fundus reflected in learning fundus images and the ageinformation of the subjects of learning fundus images.

The reception of a prediction target of fundus image 2 may includeacquiring the age data of the subject of the prediction target of fundusimage 2.

A prediction of the biometrics may include predicting the axial lengthusing the prediction target of fundus image and the age data of thesubject of the prediction target of fundus image.

The biometrics may include the fundus age.

At least one characteristic of the fundus reflected in the fundus imagesmay include at least one condition of at least one blood vessel shown inthe fundus images.

According to another aspect of the present disclosure, there is providedan apparatus for predicting biometrics using a fundus image, consistingof a learning unit for preparing a plurality of learning fundus imagesand corresponding biometric data, generating a learning model usinglearning fundus images and the biometric data based on at least onecharacteristic of the fundus reflected in the learning fundus images,and an inference unit for receiving a prediction target of fundus imageand predicting the biometrics of the subject using the generatedlearning model.

The apparatus for predicting the biometrics using a fundus imageaccording to an exemplary embodiment of the present disclosure mayfurther include a visualization unit for producing the evidence area ina prediction target of fundus image, which indicates the contribution tothe prediction of the biometric expressed as numerical values greaterthan or equal to a predetermined threshold, and outputting theemphasized evidence area in the prediction target of fundus image.

The learning unit may acquire the age data of subjects of a plurality oflearning fundus images.

The learning unit may generate a learning model for predicting thebiometrics based on at least one characteristic of the fundus reflectedin learning fundus images and the age information of the subjects of thelearning fundus images.

The inference unit may acquire the age data of the subject of aprediction target of fundus image.

The inference unit may predict the axial length using a predictiontarget of fundus image and the age data of the subject of the predictiontarget of fundus image.

The above-mentioned aspects are merely exemplary and should not beconstrued as limiting the present disclosure. In addition to theabove-described exemplary embodiments, additional exemplary embodimentsmay exist in the drawings and detailed description of the disclosure.

According to the present disclosure, it is possible to provide anapparatus and a method for predicting biometrics of a subject such asthe axial length and the fundus age, using a fundus image in the purposeof reducing any type of cost to measure the biometrics such as preparingan expensive equipment or time and space required to operate themeasuring machine.

According to the present disclosure, it is possible to support countriesthat cannot afford biometric-measuring devices or do not have developedmedical industry infrastructure, by providing a simple and stable methodto secure important biometrics such as an axial length and a fundus age.

According to the present disclosure, it is possible to secure thereliability of the predicted biometrics and assist accurate judgement ofmedical staffs who confirm the predicted biometrics by providing theemphasized evidence area expressed as numerical values indicating thedegree of contribution to the prediction of the biometrics.

The effects according to the present disclosure are not limited to thecontents exemplified above, and more various effects are included in thepresent specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and other advantages of thepresent disclosure will be more clearly understood from the followingdetailed description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a schematic configuration diagram of a biometric systemincluding an apparatus for predicting biometrics using a fundus imageaccording to an exemplary embodiment of the present disclosure.

FIG. 2 is a schematic diagram for describing the axial length.

FIG. 3 is a schematic diagram for describing the shape of the choroidalvessels and the optic disc, which are characteristics of the fundus in afundus image used for the biometric predictions.

FIG. 4 is a schematic diagram for describing a deep neural network-basedlearning model according to an exemplary embodiment of the presentdisclosure.

FIG. 5 is a diagram exemplarily illustrating an output of the emphasizedevidence area in a prediction target of fundus image, and each point inthe emphasized evidence area has a numerical value equal to or greaterthan a predetermined level, indicating the degree of contribution to theprediction of biometric.

FIG. 6 is a schematic configuration diagram of an apparatus forpredicting biometrics using a fundus image according to an exemplaryembodiment of the present disclosure.

FIG. 7 is an operation flowchart of a method for predicting biometricsusing a fundus image according to an exemplary embodiment of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT

Hereinafter, exemplary embodiments of the present disclosure will bedescribed in detail so as to be easily implemented by those skilled inthe art, with reference to the accompanying drawings. However, thepresent disclosure may be embodied in many different forms and is notlimited to the exemplary embodiments to be described herein. Inaddition, parts not related to the description have been omitted inorder to clearly describe the present disclosure in the drawings andthroughout this specification, like reference numerals designate likeelements.

Further, throughout this specification, when a certain part is“connected” with the other part, it is meant that the certain part maybe “directly connected” with the other part and “electrically connected”or “indirectly connected” with the other part with another elementinterposed therebetween.

Throughout this specification, it will be understood that when a memberis referred to as being “on”, “above”, “at the top of”, “under”,“below”, and “at the bottom of” the other member, it can be directly onthe other member or intervening members may also be present.

Throughout this specification, unless explicitly described to thecontrary, a case where any part “includes” any component will beunderstood to imply the inclusion of stated components but not theexclusion of any other component.

The present disclosure relates to an apparatus and a method forpredicting biometrics based on a fundus image. Particularly, the presentdisclosure relates to an apparatus and a method for predictingbiometrics such as the axial length, the fundus age, and the like whichare important for early diagnosis of ophthalmic diseases and braindiseases, using a fundus image and a deep neural network-based learningmodel.

FIG. 1 is a schematic configuration diagram of a biometric systemincluding an apparatus for predicting biometrics using a fundus imageaccording to an exemplary embodiment of the present disclosure.

Referring to FIG. 1 , a biometric system 10 includes an apparatus 100for predicting biometrics using a fundus image according to an exemplaryembodiment of the present disclosure (hereinafter, referred to as a‘biometric prediction apparatus 100’), a fundus image DB 200, and afundus image photographing device 30.

The biometric prediction apparatus 100 may communicate with a fundusimage DB 200 and a fundus image photographing device 30 via a network20. The network 20 refers to a connection structure capable ofexchanging information between nodes, such as terminals and servers.Examples of the network 20 include a 3rd generation partnership project(3GPP) network, a long term evolution (LTE) network, a 5G network, aworld interoperability for microwave access (WiMAX) network, Internet, alocal area network (LAN), a wireless local area network (Wireless LAN),a wide area network (WAN), a personal area network (PAN), a WiFinetwork, a Bluetooth network, a satellite broadcasting network, ananalog broadcasting network, a digital multimedia broadcasting (DMB)network, and the like, but are not limited thereto.

According to the exemplary embodiment of the present disclosure, thefundus image photographing device 30 may include a user terminal 31. Theuser terminal 31 may include, for example, a smartphone, a SmartPad, atablet PC, etc., and all types of wireless communication devices, suchas personal communication system (PCS), global system for mobilecommunication (GSM), personal digital cellular (PDC), personalhandyphone system (PHS), personal digital assistant (PDA), internationalmobile telecommunication (IMT)-2000, code division multiple access(CDMA)-2000, W-code division multiple access (W-CDMA), and wirelessbroadband Internet (Wibro). As another example, the fundus imagephotographing device 30 may include a fundus camera 32 which is providedin ophthalmology, university hospitals, etc. and is a dedicatedphotographing device for acquiring a fundus image of a patient.

In addition, according to the exemplary embodiment of the presentdisclosure, a fundus image DB 200 may be a database that can receive,send, and save a plurality of learning fundus images and any type ofbiometric data including the axial length and the age of subject. 2. Inthe description of the exemplary embodiment of the present disclosure,the ‘learning fundus images and a corresponding biometric data’ 1 mayrefer to training data used in a process of generating a learning modelby the biometric prediction apparatus 100 of the present disclosure, andthe ‘prediction target of fundus image’ 2 may refer to a fundus imagedata used for predicting biometrics with the generated learning model.

First, the biometric prediction apparatus 100 may prepare a plurality oflearning fundus images and a corresponding biometric data 1.Specifically, the biometric prediction apparatus 100 may receive aplurality of learning fundus images and a corresponding biometric data1, from the fundus image DB 200.

In addition, the biometric prediction apparatus 100 may adjust the datadistribution of the plurality of received learning fundus images and thecorresponding biometric data 1 to prevent overfitting when implementinga learning model to be described below. In other words, the biometricprediction apparatus 100 selects some of the plurality of receivedlearning fundus images and the corresponding biometric data 1 based on apredetermined criteria, and the selected learning fundus images and thecorresponding biometric data 1 may be used for training a learningmodel.

According to the exemplary embodiment of the present disclosure, theplurality of learning fundus images and the corresponding biometric data1 may include the corresponding age data as an additional data used fortraining a learning model for predicting the axial length, and all thementioned data can be pre-stored in the fundus image 200. In otherwords, when the biometric prediction apparatus 100 receives a pluralityof learning fundus images and the corresponding axial length data as thelabel to train a learning model which predicts the axial length, thebiometric prediction apparatus 100 may acquire, the age data of thesubjects as an additional information for the training.

After training a learning model, the biometric prediction apparatus 100can collect an additional plurality of learning fundus images and thecorresponding axial length data 1 based on the errors in predictions ofthe learning model, and re-train the learning model with the newlycollected data to increase the performance in certain prediction region.

In addition, the biometric prediction apparatus 100 may generate alearning model based on at least one characteristic of the fundusreflected in learning fundus image. In addition, the biometricprediction apparatus 100 may generate a learning model based on at leastone characteristic of the fundus reflected in learning fundus image, andthe age information of the subjects of the plurality of learning fundusimages according to the exemplary embodiment of the present disclosure.

Here, the biometrics may include the axial length (AL). In other words,the biometric prediction apparatus 100 may generate a learning model forpredicting the axial length of a subject using his or her fundus image.

According to the exemplary embodiment of the present disclosure,referring to FIG. 1 , the biometric prediction apparatus 100 may receivea plurality of prediction target of fundus image 2 from the fundus imagephotographing device 30, but is not limited thereto. As another example,the biometric prediction apparatus 100 may be mounted (installed) on theuser terminal 31 in the form of an application. At this time, thebiometric prediction apparatus 100 has the learning model disclosedherein through self-edge learning in the user terminal 31 which is anedge device. Even if the prediction target of fundus image 2 taken atthe user terminal 31 cannot be transmitted to a separate server orcomputing device via the network, the biometric prediction apparatus 100may operate to output predicted biometrics at the user terminal 31.

In addition, the biometric prediction apparatus 100 may acquire the agedata of the subject of a prediction target of fundus image 2 whenreceiving a prediction target of fundus image 2. In addition, thebiometric prediction apparatus 100 may predict the axial length usingthe acquired prediction target of fundus image 2 and the age data of thesubject of a prediction target of fundus image 2. As such, when the agedata of the subject of a prediction target of fundus image 2 is secured,a correlation between the age and the axial length may be additionallyconsidered, so that a prediction result of the axial length may be moreaccurate.

FIG. 2 is a schematic diagram for describing the axial length.

Referring to FIG. 2 , the axial length may refer to a distance from thecornea of to the center of the macula (central fovea).

In this regard, when the biometric prediction apparatus 100 uses theaxial length as the biometrics to be predicted, specifically, thelearning model generated by the biometric prediction apparatus 100 maybe constructed to predict the axial length, based on the shapes of thechoroidal vessels and the optic disc which are characteristics of thefundus reflected in learning fundus image.

FIG. 3 is a schematic diagram for describing the shape of choroidalvessels and the optic disc, which are characteristics of the fundus in afundus image considered to predict biometrics.

In particular, FIG. 3A shows an example of the shape of choroidalvessels, which is one of the characteristics of the fundus in the fundusimage, and FIG. 3B may illustrate an example of the shape of the opticdisc which is one of characteristics of the fundus in the fundus image.

Specifically, the shape of choroidal vessels may mean the choroidalvessels reflected in a fundus image. In this regard, generally, as theaxial length is increases, the choroidal vessel area may besignificantly shown in a fundus image, and as the age of the subjectsincreases, a larger choroidal vessel area may be shown in the fundusimage, which is associated with a pigment decrease of the choroid andthe thinning process of retina. In other words, the better the choroidalvessels are observed in a fundus image, the higher the age of thesubject or the longer the axial length of the subject can be judged.

That is, when a learning model constructed using a plurality of learningfundus images and a corresponding biometric data 1 receives a predictiontarget of fundus image 2, the learning model may output longer axiallength as the choroidal vessels in the prediction target of fundus image2 are well observed or the area of the choroidal vessels issignificantly large in the prediction target of fundus image 2.

In addition, the characteristics of the optic disc, where the opticnerves of the retina are collected and the axon of ganglion cells exitsthe retina, may include the optic disc tilt, the position (coordinatesin the image) of the optic disc, the distance from the fovea, any shapeinformation such as radius, diameter, and edge length of the optic discarea, the lamina cribrosa thickness, the prelaminar tissue thickness,and anterior laminar displacement of the optic disc, and the like, whichare reflected in the fundus image.

In addition, according to the exemplary embodiment of the presentdisclosure, the biometrics may include the fundus age. In other words,the biometric prediction apparatus 100 may generate a learning model forpredicting the fundus age of the subject using the fundus images and thecorresponding age data.

In the description of the exemplary embodiment of the presentdisclosure, the fundus age means the age of a subject to be predictedusing his or her fundus image, and may be used as an index thatcomprehensively reflects the eye health condition such as the conditionof blood vessels in the fundus, and the like of the subject inferredfrom the fundus image. Accordingly, as described below, the fundus agemay be used to evaluate the health condition of a subject or to diagnose(determine) the presence or absence of a disease in a subject bycomparing the actual age of the subject and the predicted fundus age.

Specifically, when the fundus age of a subject predicted by thebiometric prediction apparatus 100 using the received prediction targetof fundus image 2 is lower than the actual age of the subject, it isdetermined that the eye health condition or systemic blood vesselcondition of the subject is good. However, conversely, when thepredicted fundus age is higher than the actual age of the subject, itmay infer that the eye health condition or systemic blood vesselcondition of the subject is abnormal.

In this regard, when the biometric prediction apparatus 100 uses thefundus age as a biometric to be predicted, specifically, a learningmodel generated by the biometric prediction apparatus 100 may beconstructed to predict the fundus age based on at least one condition ofthe fundus reflected in learning fundus images. Here, at least onecondition of the fundus may include the information of the distribution,thickness, and color of the blood vessels shown in the fundus image, andthe information of the color and contrast of the background of thefundus.

According to the exemplary embodiment of the present disclosure, thebiometric prediction apparatus 100 may construct a deep neural network(DNN)-based learning model as the learning model for predicting thebiometrics. The biometric prediction apparatus 100 may construct alearning model by referring to existing models such as LeNet, AlexNet,GoogleNet, VGGNet, ResNet, etc., which are deep neural network modelsfor image processing, but is not limited thereto. Illustratively, thetype of learning model disclosed herein may be ResNet-18.

FIG. 4 is a schematic diagram for describing a deep neural network-basedlearning model according to an exemplary embodiment of the presentdisclosure.

Referring to FIG. 4 , the deep neural network-based learning modelaccording to an exemplary embodiment of the present disclosure is amulti-layered network, and consists of four parts: the input layer, thefeature extraction part composed of convolution layers, regression partcomposed of fully connected layers, and the output layer.

In this case, the number of nodes in the input layer may be thedimension of a fundus image to input, and the number of nodes in theoutput layer may be the dimension of a biometric data. That is, thenumber of input nodes for a learning model constructed by the biometricprediction apparatus 100 disclosed herein may be the number of pixels ofthe fundus image, and the number of output nodes may be one because avalue of the axial length or a value of the fundus age is scalar.

A convolution layer is a layer that calculates the correlation betweenthe local characteristics of the input image (the characteristics of alocal area) and the specific patterns, and the feature extraction partmay consist of multiple convolution layers, and properties of eachconvolution layer may be the number of filters, stride and type ofconvolution operation. As such, in the feature extraction partconsisting of multiple convolution layers, the level of abstraction isdetermined according to the number of convolution layers, and as thenumber of layers increases (in other words, as the layers get deeper),detailed features may be extracted from the input image, generally.According to the exemplary embodiment of the present disclosure, inorder to improve efficiency and accuracy of extracting localcharacteristics, multiple convolution layers can be connected inparallel or sequential, and any other options for setting a featureextraction part may be additionally provided. In addition, pooling, anonlinear filter, and the like that extract only necessary valuesbetween convolution layers may be applied to reduce noise or unnecessarysignal from the previous step.

Based on the feature vectors extracted from a feature extraction part,in the regression part, multiple fully connected layers may besequentially connected, ending with a node that finally outputs apredicted value (scalar or result value), after performing regressionthrough the multiple fully connected layers.

In addition, the biometric prediction apparatus 100 may produce theevidence area in a prediction target of fundus image 2, where each pointhas a numerical value equal to or greater than a predeterminedthreshold, indicating the degree of contribution to the prediction ofbiometrics (e.g., axial length, fundus age, etc.).

In addition, the biometric prediction apparatus 100 may output (display)the emphasized evidence area in a prediction target of fundus image 2.For example, the biometric prediction apparatus 100 makes the color,contrast, sharpness, etc. of the produced evidence area distinguishedfrom the background areas in a prediction target of fundus image 2, sothat users (medical staff, etc.) can visually identify or recognize theevidence area promptly.

In addition, according to the exemplary embodiment of the presentdisclosure, as the change of the calculated values of points (pixels) inan emphasized evidence area is abrupt, the changes in hue, brightness,and saturation of the pixels can be made significant.

FIG. 5 is a diagram exemplarily illustrating an output of the emphasizedevidence area in a prediction target of fundus image 2, where each pointin the emphasized evidence area has a numerical value equal to orgreater than a predetermined threshold, indicating the degree ofcontribution to the prediction of biometric.

According to the exemplary embodiment of the present disclosure, thebiometric prediction apparatus 100 applies a differential operation tothe output with respect to the nodes in the input layer or the nodes ofthe last layer of the feature extraction part to evaluate thecontribution to the predicted value of biometrics.

In particular, FIG. 5A exemplarily illustrates a prediction target offundus image 2, FIG. 5B exemplarily illustrates a prediction target offundus image 2 with the emphasized evidence area after calculating theevidence area by an application of differentiation of the output withrespect to the input, and FIG. 5C exemplarily illustrates a predictiontarget of fundus image 2 with the emphasized evidence area aftercalculating the evidence area using an application of differentiation ofthe output with respect to the nodes of the last convolutional layer ofthe feature extraction part. In addition, according to the exemplaryembodiment of the present disclosure, FIG. 5C illustrating an example ofthe emphasized evidence area in a prediction target of fundus image 2may be referred to as saliency map, class activation map (CAM), andgradient-class activation map (Grad-CAM), and the like depending on themethod used.

Hereinafter, with reference to Equations 1 to 4, an exemplary embodimentin which the biometric prediction apparatus 100 produces the evidencearea in a prediction target of fundus image 2 will be described indetail.

Using a backpropagation algorithm, the parameters of a deep neuralnetwork-based learning model described in FIG. 4 can be optimized, and atrained learning model can be used for producing the estimates of thebiometrics (e.g., axial length, fundus age, etc.) as well as theevidence area computed by differentiating the output (the estimates)with respect to the nodes of the input layer, or with respect to thenodes of the last layer of the feature extraction part.

Specifically, when a learning model is a deep neural network consistingof n fully connected layers in the regression part, and m convolutionallayers in the feature extraction part, the learning model may be definedas a function in a specific function space as shown in Equation 1 below.

Y=f _(fc) _(n) . . . f _(fc) ₂ f _(fc) ₁ (f _(cv) _(m) . . . f _(cv) ₂ f_(cv) ₁ (X _(input))   [Equation 1]

In addition, it is possible to know which nodes are important inpredictions that a neural network-based learning model makes, byobserving a change in the value of the output node according to a changein the value of a node of interest, which may be a part of calculatingthe degree of contribution to the output for each node, and thiscalculation may be a part of a process for obtaining the evidence areausing the input layer or the last layer in the feature extraction partas expressed in Equation 2 to 4 below.

X′=f _(cv) _(m) . . . f _(cv) ₂ f _(cv) ₁ (X _(input))   [Equation 2]

In addition, it is possible to obtain the derivative of the outputnumerically with respect to a specific node, X′_(i,i,j,) at i'th row andj'th column of the k'th channel, X′_(k), among the channels (filters) inthe layer, X′, and it could be a way to numerically calculate the degreeof contribution to the output for a node of specific channel in a layer,by multiplying the obtained differential value as a weight by the valueof the node. In addition, it could be a way to numerically calculate thedegree of contribution of to the output for the nodes at the samecoordinate of channels in a layer, by adding all calculatedcontributions of the nodes, and, in this way, it is possible to evaluatethe contribution that an area, which is made up of multiple nodes in thelast convolution layer of the feature extraction part, makes to theprediction of a learning model, as expressed in Equation 3 below.

$\begin{matrix}{{Y_{{reason},i,j} = {W_{i,j} \cdot X_{i,j}^{\prime}}},} & \lbrack {{Equation}3} \rbrack\end{matrix}$ $W_{i,j} = \frac{\partial Y}{\partial X_{i,j}^{\prime}}$$Y_{{reason},i,j} = {\sum\limits_{k}{W_{k,i,j} \cdot X_{k,i,j}^{\prime}}}$

-   -   where k is the index of the channels in the last layer of the        feature extraction part

In addition, in relation to the producing the evidence area using theinput layer, the derivative of the output with respect to the nodes inthe input layer may be used as a weight when calculating the degree ofcontribution to the output, and a possible form is expressed in Equation4 below.

$\begin{matrix}{Y_{{reason},i,j} = {\sum\limits_{input}{W_{{input},i,j} \cdot X_{{input},i,j}^{\prime}}}} & \lbrack {{Equation}4} \rbrack\end{matrix}$

As such, the biometric prediction apparatus 100 disclosed hereinproduces the evidence area where the degree of the contribution to aprediction and display with an emphasis, thereby securing thereliability of the predicted biometrics (e.g., axial length, fundus age,etc.) and assisting the accurate decision of medical staffs who confirmthe predictions.

FIG. 6 is a schematic configuration diagram of an apparatus forpredicting biometrics using a fundus image according to an exemplaryembodiment of the present disclosure.

Referring to FIG. 6 , the biometric prediction apparatus 100 may includea learning unit 110, an inference unit 120, and a visualization unit130.

The learning unit 110 may prepare a plurality of learning fundus imagesand a corresponding biometric data 1. In addition, the learning unit 110may acquire the age data of the subjects of a plurality of learningfundus images.

In addition, the learning unit 110 may generate a learning model usingthe prepared fundus images and the corresponding biometric data, basedon at least one characteristic of the fundus reflected in the preparedplurality of learning fundus images. In addition, the learning unit 110may use the age data as an additional information for generating alearning model which predicts biometrics based on at least onecharacteristic of the fundus reflected in the plurality of learningfundus images. In other words, the learning unit 110 may train alearning model using not only the plurality of learning fundus image andthe corresponding biometric data 1 as the input and the label, but alsothe age data as an additional information of the subject of theplurality of learning fundus images.

Here, the biometrics may include the axial length and the fundus age. Inaddition, at least one characteristic of the fundus reflected in afundus image may include the shape of the choroidal vessels and theoptic disc, and any condition of the fundus shown in a fundus image.

The inference unit 120 may receive a prediction target of fundus image2. In addition, the inference unit 120 may acquire the age data of thesubject of a prediction target of fundus image 2.

In addition, the inference unit 120 may predict biometrics of thesubject using a prediction target of fundus image 2 and the learningmodel generated by the learning unit 110. In addition, the inferenceunit 120 may predict the axial length using a prediction target offundus image 2 with the age data of the subject of the prediction targetof fundus image 2.

The visualization unit 130 may produce the emphasized evidence area in aprediction target of fundus image 2, where each point in the emphasizedevidence area has a numerical value equal to or greater than apredetermined threshold, indicating the degree of contribution to theprediction of biometric.

According to the exemplary embodiment of the present disclosure, thevisualization unit 130 may display an image (e.g., saliency map, CAM(Class activation map), Grad-CAM(gradient-class activation map), etc.that may be understood through FIG. 5C) showing the emphasized evidencearea in a prediction target of fundus image 2 itself or together withthe prediction target of fundus image 2 (e.g., displayed in parallelhorizontally or vertically), but the present disclosure is not limitedthereto. As another example, the visualization unit 130 may providemultiple modes for displaying the emphasized evidence area in aprediction target of fundus image 2, and may operate to switch each modeaccording to an input to the biometric prediction apparatus 100.

Hereinafter, an operational flow of the present disclosure will bebriefly described based on the contents described above in detail.

FIG. 7 is an operation flowchart of a method for generating a learningmodel with fundus images and corresponding biometric data, andpredicting biometrics using a fundus image according to an exemplaryembodiment of the present disclosure.

The method for predicting biometrics using a fundus image illustrated inFIG. 7 may be performed by the biometric prediction apparatus 100described above. Accordingly, even if omitted below, the contentsdescribed for the biometric prediction apparatus 100 may be equallyapplied to the description of the method for predicting biometrics usinga fundus image.

Referring to FIG. 7 , in step S11, the learning unit 110 may prepare aplurality of learning fundus images and a corresponding biometric data1.

In addition, in step S11, the learning unit 110 may acquire a pluralityof learning fundus images and a corresponding biometric data 1 and theage data of the subjects of a plurality of learning fundus images.

In addition, in step S12, the learning unit 110 may generate a learningmodel using the prepared plurality of learning fundus images and thecorresponding biometric data 1 as the input and the label, based on atleast one characteristic of the fundus reflected in the preparedplurality of learning fundus images.

In addition, in step S12, the learning unit 110 may generate a learningmodel for predicting biometrics based on at least one characteristic ofthe fundus reflected in the fundus image, and the age information of thesubjects the prepared plurality of learning fundus images.

Next, in step S13, the inference unit 120 may receive a predictiontarget of fundus image 2.

In addition, in step S13, the inference unit 120 may receive the agedata of the subject of a prediction target of fundus image 2.

In addition, in step S14, the inference unit 120 may predict biometricsof the subject using his or her prediction target of fundus image 2 anda learning model generated in step S12.

In addition, in step S14, the inference unit 120 may predict an axiallength as a biometric using the prepared prediction target of fundusimage 2 with age data of the subject of a prediction target of fundusimage 2 acquired in step S11.

Next, in step S15, the visualization unit 130 may produce the evidencearea where each point (pixel) in the evidence area has a value equal toor greater than a predetermined threshold, indicating the degree ofcontribution to the prediction of biometrics.

Next, in step S16, the visualization unit 130 may output the emphasizedevidence area in a prediction target fundus image 2 after producing theevidence area in step S15.

In the above description, steps S11 to S16 may be subdivided into moresteps or may be merged into fewer steps according to an exemplaryembodiment of the present disclosure. In addition, some steps may alsobe omitted if necessary, or the order between the steps may also bechanged.

The method for predicting biometrics based on a fundus image accordingto the exemplary embodiment of the present disclosure may be implementedin a form of program instructions which may be performed through variouscomputer means to be recorded in a computer readable medium. Thecomputer readable medium may include program instructions, data files,data structures, and the like alone or in combination. The programinstructions recorded in the medium may be specially designed andconfigured for the present disclosure, or may be publicly known to andused by those skilled in the computer software art. Examples of thecomputer readable medium include magnetic media, such as a hard disk, afloppy disk, and a magnetic tape, optical media such as a CD-ROM and aDVD, magneto-optical media such as a floptical disk, and hardwaredevices such as a ROM, a RAM, and a flash memory, which are speciallyconfigured to store and execute the program instructions. Examples ofthe program instructions include high language codes executable by acomputer using an interpreter and the like, as well as machine languagecodes created by a compiler. The hardware device may be configured to beoperated with one or more software modules in order to perform theoperation of the present disclosure and vice versa.

Further, the aforementioned method for predicting biometrics using afundus image may be implemented even in a form of computer programs orapplications to be executed by a computer, which are stored in therecording medium.

The aforementioned description of the present disclosure is to beexemplified, and it can be understood by those skilled in the art thatthe technical spirit or required features of the present disclosure canbe easily modified in other detailed forms without changing. Therefore,it should be appreciated that the embodiments described above areillustrative in all aspects and are not restricted. For example,respective components described as single types can be distributed andimplemented, and similarly, components described to be distributed canalso be implemented in a coupled form.

The scope of the present disclosure is represented by claims to bedescribed below rather than the detailed description, and it is to beinterpreted that the meaning and scope of the claims and all the changesor modified forms derived from the equivalents thereof come within thescope of the present disclosure.

What is claimed is:
 1. A method for predicting biometrics using a fundusimage comprising following steps: preparing a plurality of learningfundus images; generating a learning model for predicting correspondingbiometrics using the prepared fundus images, based on at least onecharacteristic of the fundus reflected in the prepared plurality oflearning fundus images; receiving a prediction target of fundus image;and predicting the biometrics of the subject of a prediction target offundus image by using the generated learning model.
 2. The method forpredicting biometrics according to claim 1, wherein the biometricsinclude the axial length, and the at least one characteristic of thefundus reflected in a fundus image includes the shape of a choroidalvessels and the optic disc.
 3. The method for predicting biometricsaccording to claim 2, further comprising: Producing the evidence areaexpressed as numerical values greater than or equal to a predeterminedthreshold, which represents the degree of contribution to the predictionof the biometrics in a prediction target of fundus image.
 4. The methodfor predicting biometrics according to claim 3, further comprising:outputting the emphasized evidence areas in a prediction target offundus image.
 5. The method for predicting biometrics according to claim2, wherein the preparation of a plurality of learning fundus imagesincludes acquiring the age data of the subjects of a plurality oflearning fundus images, and the generation of a learning model includesgenerating a learning model for predicting biometrics using the fundusimages, based on at least one characteristic of the fundus reflected inthe fundus image and the age information of the subjects of theplurality of learning fundus images.
 6. The method for predictingbiometrics according to claim 5, wherein the reception of a predictiontarget of fundus image includes acquiring the age data of the subject ofthe prediction target of fundus image, and the prediction of thebiometrics includes predicting the axial length using a predictiontarget of fundus image with the age data of the subject of theprediction target of fundus image.
 7. The method for predictingbiometrics according to claim 1, wherein the biometrics include thefundus age, and the at least one characteristic of the fundus reflectedin a fundus image includes at least one condition of at least one bloodvessel shown in the fundus image.
 8. An apparatus for predictingbiometrics based on a fundus image comprising: a learning unit forpreparing a plurality of learning fundus images, and generating alearning model for predicting corresponding biometrics using theprepared fundus images, based on at least one characteristic of thefundus reflected in the plurality of learning fundus images; and aninference unit for receiving a prediction target of fundus image andpredicting biometric of the subject of a prediction target of fundusimage using the generated learning model.
 9. The apparatus forpredicting biometrics according to claim 8, wherein the biometricsinclude an axial length, and at least one characteristic of the fundusreflected in the fundus image includes the shape of choroidal vesselsand the optic disc.
 10. The apparatus for predicting biometricsaccording to claim 9, further comprising: a visualization unit forproducing the evidence area in a prediction target of fundus image,expressed as numerical values greater than or equal to a predeterminedthreshold, which represents the degree of contribution to prediction ofthe biometrics, and outputting the evidence area with an emphasis in theprediction target of fundus image.
 11. The apparatus for predictingbiometrics according to claim 9, wherein the learning unit acquires aplurality of learning fundus images, and the age data of the subjects ofa plurality of learning fundus images, and generates a learning modelfor predicting biometrics based on at least one characteristic of thefundus reflected in the fundus image and the age information of thesubjects of the plurality of learning fundus images.
 12. The apparatusfor predicting biometrics according to claim 11, wherein the inferenceunit acquires a prediction target of fundus image and the age data ofthe subject of a prediction target of fundus image and predicts theaxial length using the prediction target of fundus images and the agedata of the subject of the prediction target of fundus image.
 13. Theapparatus for predicting biometrics according to claim 8, wherein thebiometrics include the fundus age, and at least one characteristic ofthe fundus reflected in a fundus image includes a condition of at leastone blood vessel shown in the fundus image.
 14. A non-transitorycomputer readable recording medium which records programs for executingthe method of claim 1 in a computer.